System and method for music education

ABSTRACT

A system for music education includes a storage device being configured to store a set of record data; a recording device being configured to record a musical exercise; a processor being connected to the storage device and the recording device, and configured to determine the start and the end of the musical exercise by the detected volume of the musical exercise, to convert the recording of the musical exercise to a set of user data, to retrieve the set of record data and to map the set of user data to the set of record data, to analyze differences between the two based on the mapping and to calculate a series of values and to calculate a score as the sum of the series of values multiplied by a series of coefficients respectively; and an electronic display being connected to the processor. A method for music education is also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/507,608, filed on Jul. 14, 2011; the contents of which is herebyincorporated by reference.

FIELD OF THE PATENT APPLICATION

The present patent application generally relates to computer assistedmusic education and more specifically to a system and a method for musiceducation.

BACKGROUND

One of the most important trainings in music education is to improvemusicianship of students. Musicianship includes the capabilities ofsinging and playing music instrument with correct frequencies andtiming, counting beat correctly and finding characters of music byhearing. These are also one of the major subjects in music examinationsprovided by international music institutes, such as Associate Board ofRoyal School Music (ABRSM) in UK, Royal Conservatory of MusicExaminations (RCM) in Canada, Australian Music Examinations Board (AMEB)in Australia and etc.

Previously, all these trainings can only be provided by teachers withina room. While there are some other materials, e.g. recorded waves in CDor online to train students in finding characters of music by hearing,there is no way for students to learn singing, playing a musicinstrument, and counting beats without the teacher as the performancemust be judged by the teacher and the teacher will give feedback afterlistening to the student's performance. Even though students can recordthe performance and send the recording to the teacher though a network,it still requires a teacher to judge.

On the other hand, in any teaching with a teacher, the teacher can onlytell what the good points and bad points about a student's performanceare by mouth. The teacher can record the performance and replay it.However, it is very inconvenient for the teacher to record theperformance every time. Even when there is a recording, the studentneeds to imagine which part of the performance is good or bad based onteacher's words, without any visual aids to help.

In addition, since the performance of the students can vary in manydirections, the teacher cannot record the performance in a scientificway. In other words, the teacher can only mark some major errors but itis difficult for the teacher to have a whole picture on the progress ofthe students' performance.

SUMMARY

The present patent application is directed to a system and a method formusic education. In one aspect, the system includes a storage devicebeing configured to store a set of record data that contains informationabout speed, timing, frequency and beats of a standard musical sample; arecording device being configured to record a musical exercise; aprocessor being connected to the storage device and the recordingdevice, and configured to determine the start and the end of the musicalexercise by the detected volume of the musical exercise, to convert therecording of the musical exercise to a set of user data, to retrieve theset of record data from the storage device and to map the set of userdata to the set of record data, to analyze differences between the userdata and the record data based on the mapping and thereby to calculate aseries of values related to speed, timing, frequency or power of theuser data, and to calculate a score as the sum of the series of valuesmultiplied by a series of coefficients respectively; and an electronicdisplay being connected to the processor and configured to display theseries of values, the score, or the differences between the user dataand the record data to a user.

In another aspect, the present patent application provides acomputer-implemented method for music education. The method includesstoring a set of record data that contains information about speed,timing, frequency and beats of a standard musical sample on a storagedevice; recording a musical exercise with a recording device whiledetermining the start and the end of the musical exercise by thedetected volume of the musical exercise with a processor; converting therecording of the musical exercise to a set of user data with theprocessor; retrieving the set of record data from the storage device andmapping the set of user data to the set of record data with theprocessor; analyzing differences between the user data and the recorddata based on the mapping and thereby calculating a series of valuesrelated to speed, timing, frequency or power of the user data with theprocessor; calculating a score as the sum of the series of valuesmultiplied by a series of coefficients respectively with the processor;and displaying the score and the differences between the user data andthe record data to a user through an electronic display.

Recording the musical exercise may include starting recording by theprocessor when the detected volume of the musical exercise has beenhigher than a predetermined threshold for a predetermined number oftimes, and ending recording by the processor when the detected volume ofthe musical exercise has been lower than a predetermined threshold for apredetermined number of times.

The musical exercise may include a sequence of clapping, and convertingthe recording of the musical exercise to the set of user data mayinclude calculating the sum of the peak power of the clapping multipliedby a first coefficient and the total power of the clapping multiplied bya second coefficient, and thereby detecting the sequence of clappingfrom the set of user data and eliminating noise from the result of thedetection by the processor.

The method may further include calculating the duration of each clap inthe adjusted user data and in the record data respectively, anddetermining whether the difference therebetween is within apredetermined tolerance by the processor.

The musical exercise may include singing or music instrument playing,and the method may further include filtering out any music note that isin a predetermined frequency in the user data and adjusting the scale ofthe user data according to the scale of the record data before themapping by the processor.

The method may further include transforming the record data and the userdata into a matrix, calculating a cost of error for each node of thematrix, and adjusting the speed and the key signature of the record databy finding a path with the lowest accumulative cost of error based ondifferent combination of speeds and key signatures by the processor.

In yet another aspect, the present patent application provides acomputer-implemented method for music education. The method includesstoring a set of record data that contains information about speed,timing, frequency and beats of a standard musical sample on a storagedevice; recording a musical exercise with a recording device; convertingthe recording of the musical exercise to a set of user data with aprocessor; retrieving the set of record data from the storage device andmapping the set of user data to the set of record data with theprocessor; comparing the user data to the record data based on themapping with the processor; and displaying the difference between theuser data and the record data as well as a score representing aquantitative evaluation of the musical exercise based on the comparisonto a user through an electronic display. Musical elements in the recorddata is displayed in a first type of color, musical elements in the userdata with a strong power is displayed in a second type of color, andmusical elements in the user data with a weak power is displayed in athird type of color.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1 is a flow chart illustrating the steps of a method for musiceducation according to an embodiment of the present patent application.

FIG. 2 is a flow chart illustrating the start and the end detection inthe method illustrated in FIG. 1.

FIG. 3 is a flow chart of conversion of recorded wave to clapping rawdata in the method illustrated in FIG. 1.

FIG. 4 is a flow chart of Clapping Mapping in the method illustrated inFIG. 1.

FIG. 5 is a flow chart of conversion of recorded wave to music note rawdata in the method illustrated in FIG. 1.

FIG. 6 is a flow chart of pre-mapping filtering and adjustment in themethod illustrated in FIG. 1.

FIG. 7 is a flow chart for finding whether the music note is referencepoint in the method illustrated in FIG. 1.

FIG. 8 is a flow chart of fine speed deviation calculation and timeerror checking in the method illustrated in FIG. 1.

FIG. 9 is a first flow chart of Clapping Power Analysis in the methodillustrated in FIG. 1.

FIG. 10 is a second flow chart of Clapping Power Analysis in the methodillustrated in FIG. 1.

FIG. 11 is a demo screen showing the effect of using “overlappeddisplay” to show record data and the student's performance in clappingtraining according to an embodiment of the present patent application.

FIG. 12 is a demo screen shows the effect of using “overlapped display”to show record data and student's performance in singing/musicinstrument playing training according to an embodiment of the presentpatent application.

FIG. 13 is a demo screen shows the effect of using “dual row display” toshow record data and student's performance in clapping trainingaccording to an embodiment of the present patent application.

FIG. 14 is a Demo screen shows the effect of using “dual row display” toshow record data and student's performance in singing/music instrumentplaying training according to an embodiment of the present patentapplication.

FIG. 15 is a flow chart illustrating a server checking macro errors inthe method illustrated in FIG. 1.

DETAILED DESCRIPTION

According to an embodiment of the present patent application, a systemfor music education includes a storage device, a recording device, aprocessor and an electronic display. The storage device is configured tostore a set of record data that contains information about speed,timing, frequency and beats of a standard musical sample. The recordingdevice is configured to record a musical exercise. The processor isconnected to the storage device and the recording device, and configuredto determine the start and the end of the musical exercise by thedetected volume of the musical exercise, to convert the recording of themusical exercise to a set of user data, to retrieve the set of recorddata from the storage device and to map the set of user data to the setof record data, to analyze differences between the user data and therecord data based on the mapping and thereby to calculate a series ofvalues related to speed, timing, frequency or power of the user data,and to calculate a score representing a quantitative evaluation of themusical exercise based on the series of values. The electronic displayis connected to the processor and configured to display the series ofvalues, the score, or the differences between the user data and therecord data to a user.

FIG. 1 is a flow chart illustrating the steps of a method for musiceducation according to an embodiment of the present patent application.The blocks in FIG. 1 are described in detail hereafter.

Record Data

The key of the examination and training is to compare the performance ofstudent and the original music/beat. In this system, the originalmusic/beat is stored. The time signature, key signature, speed,tonality, absolute frequencies and beats of all music notes are stored.Such information of the original music is stored on a server or a user'sdevice in this embodiment and referred to as the Record Data hereafter.For analysis of the student's performance in singing or playing musicinstrument, the music notes and frequencies with the time stamp of themusic are used as the source for comparison. For analysis of thestudent's performance in counting beat (by clapping, making anystandardized noise or any means that can show the intention of thestudent's beat counting), the timestamp of the music note (for theanalysis of the student's performance in counting beats of each musicnotes) or timestamp calculated by time signature with speed information(for the analysis of the student's performance in counting standard beatinformation of the whole song) are used as the source for comparison.

Singing, Music Instrument Playing and Clapping Start and End Detectionby Volume

In order to check whether the user has started and stopped singing,music instrument playing and clapping so that the system can response inan interactive way, the system has an algorithm to detect the start andstop of student's performance by volume.

FIG. 2 is a flow chart illustrating the start and the end detection inthe method illustrated in FIG. 1. Referring to FIG. 1 and FIG. 2, in thesystem, two parameters, VolThreshold and EndFlag, are being updatedduring the recording, in every preset period. VolThreshold will bechanged to 1 when the recording volume reaches a threshold 1. Theanalysis system can analyze the recording to give a reasonable andcorrect result if the recording volume is higher than the threshold 1.EndFlag will be changed to 1 when the recording volume is lower thanthreshold 2. The threshold 2 must be lower than threshold 1.

During the recording, if the system can detect a predeterminedcontinuous number of EndFlag=0, the system will consider the student'sperformance has been started. The recording will be started, no matterwhether this signal is detected. If the system cannot detect a minimumnumber of EndFlag=0 within a preset period, the system will considerstudent cannot start the performance and show warning and the recordingwill be ended. After system considers the performance has been started,if the system can detect another predetermined continuous number ofEndFlag=1, the system will consider student's performance as havingended. During the recording, if there is any moment in which the systemcan detect a minimum number of VolThreshold=1, the system will considerthe recording is valid. Otherwise, after the recording, the system willprompt that the recording is not valid due to low volume of thestudent's performance

Conversion of Recorded Wave to Clapping Raw Data

FIG. 3 is a flow chart of conversion of recorded wave to clapping rawdata in the method illustrated in FIG. 1. There are 2 types of clappingtraining. The first type is clapping alone without any music backgroundand the second type is clapping when music is being played. In thesecond type, the clapping sound is mixed with the music in audiorecording.

The conversion can handle both types of clapping training. The algorithmis described hereafter.

Clapping Detection from Clapping Raw Data

A list of clapping with the timestamp, the total power of the clappingand the peak power of the clapping are provided by the front end engine.However, noises are also included in the output. The system will use thefollowing algorithm to eliminate the noise.

Clapping will be considered as real clapping ifα_(range(x))×Peak Power+β_(range(x))×Total Power>ThresholdClapping_(range(x))

The clapping will be remained in the list, for further analysis.

Clapping will be considered as unclear clapping ifα_(range(x))×Peak Power+β_(range(x))×Total Power<ThresholdUnclear_(range(x))

The clapping will be remained in the list with a marking indicating itis unclear, for further analysis.

Clapping will be considered as noise ifα_(range(x))×Peak Power+β_(range(x))×Total Power<ThresholdUnclear_(range(x))

The clapping will be eliminated from the list, wherein, x is the rangethat the current Peak Power belongs to.

Clapping Mapping

FIG. 4 is a flow chart of Clapping Mapping in the method illustrated inFIG. 1. The list of clapping after noise elimination will be used tocompare with the record data, no matter what the beat of each note orthe standard beat information of the whole song is.

After executing this algorithm, the system will know which clapping iscorrect, which clapping is extra clapping, which clapping is missed, andwhich clapping is shifted in timing.

Clapping Beat Period Calculation Based on Timing

Referring to the block “Clapping Beat Period Calculation Based OnTiming”, the timing of each clapping is checked. Based on the speedinformation in the record data, the duration of each clap in the recorddata can be calculated. This duration is compared to the duration ofeach clapping.The duration of each clap:Duration_(Clapping(N))=Timestamp_(Clapping(N))−Timestamp_(Clapping(N+1))

If the clapping duration is within the tolerance, i.e.Duration_(Clapping(N))−Allowance₁>Duration_(Clapping(N))>Duration_(Clapping(N))+Allowance₂

The clapping will be considered as having correct beat period.

If the percentage of clapping having correct beat period>threshold beatperiod, the whole song will be marked as good beat period; if not, thewhole song will be marked as bad beat period.

Conversion of Recorded Wave to Music Note Raw Data

FIG. 5 is a flow chart of conversion of recorded wave to music note rawdata in the method illustrated in FIG. 1. This conversion is used in thesinging or musical instrument playing training.

Pre-Mapping Filtering and Adjustment

FIG. 6 is a flow chart of pre-mapping filtering and adjustment in themethod illustrated in FIG. 1. The filtering and adjustment is used insinging training. Abnormal music note raw data filtering is in thisblock. Unlike the music note range of a music instrument, a singleperson can only sing within a limited range. The purpose of thisfiltering is to eliminate abnormal music note which cannot be sung byhuman

After that, the filtered data will be checked whether its scale is thesame as the scale of the record data. This adjustment is called “globalscale frequency adjustment.”

Initial Mapping with Raw Frequency Adjustment and Raw Speed Adjustment

This mapping gives an initial linkage between the record data and thestudent's performance. It is a straight forward process if (1) thestudent's performance is perfect, (2) there is no noise during therecording, and (3) there is no error in the conversion of the recordedwaves to the music note raw data. However, there must be errors amongthese 3 factors and this mapping will take these factors intoconsideration.

In the mapping algorithm, the following terms are used

-   -   Missed note=The note that the student should sing/play but does        not sing/play during the recording    -   Extra note=The student has sung/played a note which is not in        the record data    -   Short note=Duration of the note<Threshold_(Short Note). The        duration of this note is too short and it is difficult for a        human to sing or play. Most probably it is noise during the        recording and it is not eliminated in the conversion before the        initial mapping.

The record data and the music raw data are put into a matrix. If thereare M music raw data and N record data, then an M×N matrix will beformed. A cost of error will be calculated for each matrix node, basedon the following formula.α×Absolute(Record data_(Frequency)−Music RawData_(Frequency))+β×Absolute(Record data_(Duration)−Music RawData_(Duration))+Ω×Short Note Penalty+μ×Missed Note Penalty+η×Extra NotePenalty

The frequency and duration mentioned above is absolute frequency andduration. Short Note penalty will be included if the duration of theMusic Raw data<Threshold_(Short Note). α, β, Ω, μ and η are the weightof the different factors in the formula.

After calculating the cost of error of each matrix node, the algorithmwill find the path from lowest left hand side of the matrix to thehighest right hand side of the matrix, with the lowest accumulative costof errors. Since student may start to sing/play not from the beginningof the music and may end at any point before the end of music, the pathfound can be started not from the node (0, 0) and ended not at the node(M, N). However, if in the case the path is not started from the node(0, 0) or not ended at the node (M, N), the related missed note penaltyand the extra note penalty will be added to the accumulative cost oferrors.

Since the speed and the key signature of the student's performance maybe different from the record data, the algorithm will find the path withthe lowest accumulative costs based on different combination of speedand key signature by adjusting the key signature and speed of the recorddata. The variation of the speed and the key signature can be raw, i.e.the interval between each test case can be raw as the main purpose ofthis part is to get the initial mapping information.

The path with the lowest accumulative costs will be marked as theinitial mapping path.

The record data that cannot be mapped will be marked as “Missed Note”.

The student's performance music note raw data, whoseduration<Threshold_(Short Note) and cannot be mapped, will be marked as“Short Note”.

The student's performance music note raw data, whoseduration>Threshold_(Short Note) and cannot be mapped, and whosefrequency is within N semitones of the frequency of the music note rawdata mapped before it, will be marked as “Sliding Note”.

The student's performance music note raw data, whoseduration>Threshold_(Short Note) and cannot be mapped, and whosefrequency is higher or lower than the frequency of the music note rawdata mapped before it by N semitones, will be marked as “Extra Note”.

Reference Point, Fine Frequency Deviation Calculation and FrequencyError Checking

Even student's performance music note raw data can be mapped to therecord data. There may be frequency errors and time errors. Human willconsider the music note sang/played as an error music note if thedifference between the frequency of the previous music note and thefrequency of current music note is deviated from the requirement. Eventhe absolute frequency of the music note is incorrect; human may stillthink the music note is correct. In this system, both the relativefrequency error and the absolute frequency error are considered. Forfrequency errors, there are 4 types of errors.

-   -   Relative Slight=The difference between the frequency of the        previous music note and the    -   Frequency Error frequency of the current music note is deviated        from the requirement slightly, e.g. N semitones.    -   Relative Serious=The different between the frequency of the        previous music note and the    -   Frequency Error frequency of the current music note is deviated        from the requirement seriously, e.g. N+M semitones.    -   Absolute Slight=The absolute frequency of the current music note        is deviated from the    -   Frequency Error requirement slightly, e.g. P semitones.    -   Absolute Serious=The absolute frequency of the current music        note is deviated from the    -   Frequency Error requirement seriously, e.g. P+Q semitones.

Relative and absolute frequency errors can occur together or alone atthe same music note.

In order to find out the fine frequency adjustment, a “reference pointfor frequency adjustment” is required. The frequency of the music notein the student's performance which is marked as a reference point willbe considered as correct. The difference between the frequency of thismusic note in the student's performance and the frequency of the recorddata mapped will be treated as the fine frequency adjustment for allmusic notes in student's performance. Any one of the mapped music notein the student's performance can be the reference point based on thefollowing rules. Referring to FIG. 7, each mapped music note can bedetermined on whether it is a reference point or not.

After the frequency reference point is found, the frequencies of all themusic notes of the student's performance will be adjusted based on thedifference between the frequency of the reference point music note ofthe student's performance and the frequency of the reference pointrecord data.

After the adjustment, the 4 types of errors will be calculated andrecorded.

The difference between the frequency of the reference point music noteof the student's performance and the frequency of the reference pointrecord data will also be recorded as “global key frequency adjustment”.

Fine Speed Deviation Calculation and Time Error Checking

Similarly, there are 4 types of time errors.

-   -   Relative Slight=The difference between the timestamp of the        previous music note and the    -   Time Error timestamp of the current music note is deviated from        the requirement slightly, e.g. N beat.    -   Relative Serious=The difference between the timestamp of the        previous music note and the    -   Time Error timestamp of the current music note is deviated from        the requirement seriously e.g. N+M beat.    -   Absolute Slight=The absolute timestamp of the current music note        is deviated from the    -   Time Error requirement slightly, e.g. P beat.    -   Absolute Serious=The absolute timestamp of the current music        note is deviated from the    -   Time Error requirement seriously, e.g. P+Q beat.

Relative and serious time errors can occur together or alone at the samemusic note.

Referring to FIG. 8, in order to find out the fine speed adjustment,different speeds will be tried until the one that best fits theillustrated algorithm is found.

After speed is found, the time stamp of each music note in record datawill be re-calculated based on this speed. And then the record data withnew speed will be compared to student's performance and the 4 types oftime errors will be calculated and recorded.

The speed found will also be recorded as “student's performance speed”.

Clapping, Singing/Instrument Playing Start Time and End Time Analysis

When the student starts clapping/singing/note playing and endsclapping/singing/note playing are critical factors to judge whether theclapping/singing/note playing is good or bad. In the system, the timestamp of the first “mapped”, “shifted” or “extra” user's singing (forclapping training) and the “mapped” or “extra” student'ssinging/instrument playing, whichever timestamp is sooner (or thenumerical time is lower), will be used to determine whether the starttime is correct. Similarly, the timestamp of the first “mapped”“shifted” or “extra” user clapping, whichever timestamp is sooner, willbe used to determine whether the start time is correct. Similarly, thetimestamp of the last “mapped” shifted” or “extra” user clapping,whichever timestamp is later, will be used to determine whether the endtime is correct. Similarly, the timestamp of the last “mapped”,“shifted” or “extra” user singing/instrument playing, whichevertimestamp is later, will be used to determine whether the end time iscorrect.

Singing, Music Instrument Playing and Clapping Power Calculation

Another critical factor to consider is whether the power of clapping ormusic note is correct. Student is required to clap/sing/play with acorrect power to indicate “strong” or “weak” for each clapping. Thepower of the clapping/singing/note playing is considered as “strong” if

$\frac{{Clapping}\mspace{14mu}{or}\mspace{14mu}{Music}\mspace{14mu}{Note}\mspace{14mu}{Power}\mspace{14mu}{NX}\;\alpha}{{Moving}\mspace{14mu}{Average}\mspace{14mu}\begin{pmatrix}{{{Clapping}\mspace{14mu}{or}\mspace{14mu}{Music}\mspace{14mu}{Note}\mspace{14mu} N} -} \\{{K\mspace{14mu}\ldots\mspace{14mu}{Clapping}\mspace{14mu}{or}\mspace{14mu}{Music}\mspace{14mu}{Note}\mspace{14mu} N} + M}\end{pmatrix}} > {Threshold}_{StrongBeat}$wherein α is the weight factor, K is the number of claps/music notesbetween current clapping/music note N and the first clapping/music noteused in the calculation of moving average, M is the number ofclapping/music note between current clapping/music note N and the lastclapping/music note used in the calculation of moving average. Theclapping/music note is considered as “weak” if it cannot fulfill theabove formula.

Clapping Power Analysis

-   -   There are 2 analyses. (It is called “Macro Power Error”).    -   1. Referring to FIG. 9, the first one is whether student has        clapped in a correct pattern. For example, in a 3 beat time        signature, the student is required to clap in a pattern “strong”        “weak” “weak” periodically. If the student claps in any other        pattern, it is considered as wrong.    -   2. Referring to FIG. 10, the second one is while student has        clapped in a correct pattern, whether the “strong” beat is        shifted in a number of periods.

If there is any clapping power error (i.e. a “strong power” clap isclapped as “weak power” or a “weak power” clap is clapped as “strongpower”) and there is no (1) incorrect beat clapping power pattern and(2) shifted clapping power, the clapping power error is considered asindividual and just marked individually.

Singing and Music Instrument Playing Power Analysis

The power requirement of singing and music instrument playing is not ina periodical format. The power of each music note will be compared tothe requirement of each note one by one.

Speed Trend Analysis

The last factor to be considered is while the timing of a series ofclapping/music note is correct, whether there is a trend of theclapping/singing/music note playing time, either going faster or slower.

If the following formula can be satisfied, the clapping in that periodwill be considered as “clapping/singing/music note playing speedincreasing”.(Timestamp_(User Clapping/Music Note(N))−Timestamp_(Record data(N)))<(Timestamp_(User Clapping/Music Note(N+1))−Timestamp_(Record data(N1)))<. . .<(Timestamp_(User Clapping/Music Note(N+K))−TimeStamp_(Record data(N+K))),wherein K=the number of claps/music notes to be checked.

Similarly, if the following formula can be satisfied, the clapping/musicnote in that period will be considered as “clapping/playing/music noteplaying speed decreasing”.(Timestamp_(User Clapping/Music Note(N))−Timestamp_(Record data(N)))>(Timestamp_(User Clapping/Music Note(N+1))−Timestamp_(Record data(N+1)))>. . .>(Timestamp_(User Clapping/Music Note(N+K))−Timestamp_(Record data(N+K))),wherein K=the number of claps/music notes to be checked.

It is assumed that Clapping_(N) is mapped to Record data_(N), which canbe any mapped pair.

Impression Analysis

The analysis will give a comment/impression about the overallperformance. Here is the formula of impression analysis for clappingtraining:Impression result_(Clapping) =a ₁×number of missed clappings+a ₂×numberof extra clappings+a ₃×number of shifted clappings+a ₄×percentage ofcorrect clapping period+a ₅×number of clapping power errors+a ₆×macropower errors+a ₇×number of speed trend decreasing+a ₈×number of speedtrend increasing+a ₉×deviation of start time from the requirement+a₁₀×deviation of end time from the requirement, wherein a _(x) is theweight factor. The system will classify the impression results based onthe different thresholds for clapping impression analysis.

Here is the formula of impression analysis for singing/instrumentplaying:Impression result_(Singing/Instrument Playing) =b ₁×number of missednotes+b ₂×number of extra notes+b ₃×number of slight frequency errors(both relative and absolute)+b ₄×number of serious frequency errors(both relative and absolute)+b ₅×number of slight time errors (bothrelative and absolute)+b ₆×number of serious time errors (both relativeand absolute)+b ₈×speed trend increasing+b ₉×speed trend increasing+b₁₀×deviation of speed from the requirement+b ₁₁×deviation of start timefrom the requirement+b ₁₂×deviation of end time from the requirement,wherein b _(x) is the weight factors. The system will classify theimpression results based on the different thresholds forsinging/instrument playing impression analysis.

Display Mapped Result and Analysis to Give Feedback

On top of using speech to let the student understand his performance,the analysis results and the recommendations will be displayed throughthe electronic display. The system displays both the record data andstudent's performance on a single screen of the electronic display.According to an embodiment of the present patent application, there aretwo ways of presentation.

1. Overlapped Display

The record data and student's performance are displayed on the samescore line and the notes of both of them are overlapped. There arespecial arrangements in the color of notes.

-   -   Clapping/notes of the record data will be in one type of color        and the student's performance will be in another type of color        in order to distinguish the two music lines while both of them        are shown on the same score.    -   Clapping/notes of the user's performance with “strong” power        will be indicated by one type of color and those with “weak”        power will be indicated by another type of color.

Note 110 (in clapping training) in FIG. 11, Note 210 (in singing/musicinstrument playing training) in FIG. 12 are record data in one type ofcolor, e.g. black. (All other record data are shown with same symbol inthe FIG. 11 and FIG. 12). It can be a clapping (if in clapping training,and since there is no note/frequency information of each clapping, thevertical position of the note has no meaning) or a music note (if insinging/music instrument playing training).

Note 120 (in clapping training) in FIG. 11 and Note 220 (insinging/music instrument playing training) in FIG. 12 correspond to thestudent's clapping/singing with “strong” power, being in another type ofcolor, e.g. red. (All other student's claps with “strong” power areshown with same symbol in the FIG. 11 and FIG. 12). It can be a clapping(if in clapping training and since there is no note/frequencyinformation of each clapping, the vertical position of the note has nomeaning) or a music note (if in singing/music instrument playingtraining).

Note 130 (in clapping training) in FIG. 11 and Note 230 (insinging/music instrument playing training) in FIG. 12 correspond to thestudent's clapping/singing with “weak” power, being in yet another typeof color, e.g. green. (All other student's clapping/singing with “weak”power are shown with the same symbol in FIG. 11 and FIG. 12). It can bea clapping (if in clapping training and since there is no note/frequencyinformation of each clapping, the vertical position of the note has nomeaning) or a music note (if in singing/music instrument playingtraining).

2. Dual Row Display

The record data and the student's performance can also be displayed ontwo score lines in parallel and the notes of both of them will be shownwith the same time scale. Similarly, there are special arrangements inthe color of notes as described above.

Note 310 (in clapping training) in FIG. 13 and FIG. 14 Note 410 (insinging/music instrument playing training) are record data in one typeof color, e.g. black. (All other record data are shown with the samesymbol in FIG. 13 and FIG. 14).

Note 320 (in clapping training) in FIG. 13 and Note 420 (insinging/music instrument playing training) in FIG. 14 correspond to astudent's clapping/singing with “strong” power, being in another type ofcolor, e.g. red. (All other student's clapping with “strong” power isshown with the same symbol in FIG. 13 and FIG. 14).

Note 330 (in clapping training) in FIG. 13 and Note 430 (insinging/music instrument playing training) in FIG. 14 correspond to astudent's clapping/singing with “weak” power, being in another type ofcolor, e.g. green. (All other student's clapping/singing with “weak”power is shown with same symbol in the above demo screen).

In both kinds of display methods, the system will point out the errorsfound during the above-mentioned analysis.

Macro Error Analysis

All results and the analysis results of individual question will bestored in a server. The server will analyze the answers and the resultsof all questions, and check whether there is similar trend in the errorsfound.

FIG. 15 is a flow chart illustrating a server checking macro errors inthe method illustrated in FIG. 1. For singing/instrument playingtraining, the server will also calculate the number of frequency errorsof the same frequency interval in order to find out whether the studentnormally has his errors in singing/instrument playing in a particularfrequency interval.

The above embodiments provide a method and a system that can capture thestudent's performance, no matter in singing, playing music instrument orclapping, analyze the performance, display students' performance and theanalysis on the result in the display media, and give feedback and theanalysis on the trend of the series of performance. The method and thesystem can be implemented in any digital platform with a processor,including but not limited to mobile phone, tablet PC, web, personalcomputer, digital music instrument and any other type of electronicdevices.

What is claimed is:
 1. A system for music education comprising: astorage device being configured to store a set of record data thatcontains information about speed, timing, frequency and beats of astandard musical sample; a recording device being configured to record amusical exercise; a processor being connected to the storage device andthe recording device, and configured to determine the start and the endof the musical exercise by the detected volume of the musical exercise,to convert the recording of the musical exercise to a set of user data,to retrieve the set of record data from the storage device and to mapthe set of user data to the set of record data, to analyze differencesbetween the user data and the record data based on the mapping andthereby to calculate a series of values related to speed, timing,frequency or power of the user data, and to calculate a score as the sumof the series of values multiplied by a series of coefficientsrespectively; and an electronic display being connected to the processorand configured to display the series of values, the score, or thedifferences between the user data and the record data to a user; whereinthe processor is configured to start recording when the detected volumeof the musical exercise has been higher than a predetermined thresholdfor a predetermined number of times, and to end recording when thedetected volume of the musical exercise has been lower than apredetermined threshold for a predetermined number of times.
 2. Thesystem of claim 1, wherein the musical exercise comprises a sequence ofclapping, and the processor is configured to calculate the sum of thepeak power of the clapping multiplied by a first coefficient and thetotal power of the clapping multiplied by a second coefficient, andthereby to detect the sequence of clapping from the set of user data andeliminate noise from the result of the detection.
 3. The system of claim2, wherein the processor is configured to calculate the duration of eachclap in the adjusted user data and in the record data respectively, andto determine whether the difference therebetween is within apredetermined tolerance.
 4. The system of claim 1, wherein the musicalexercise comprises singing or music instrument playing, and theprocessor is configured to filter out any music note that is in apredetermined frequency in the user data and adjust the scale of theuser data according to the scale of the record data before the mapping.5. The system of claim 4, wherein the processor is configured totransform the record data and the user data into a matrix, calculate acost of error for each node of the matrix, and adjust the speed and thekey signature of the record data by finding a path with the lowestaccumulative cost of error based on different combination of speeds andkey signatures.
 6. The system of claim 4, wherein the processor isconfigured to mark a reference point in the user data and adjust thefrequencies of all the music notes in the user data based on thedifference between the user data and the record data at the referencepoint.
 7. The system of claim 6, wherein the processor is configured tocalculate the frequency errors in the user data based on the mapping, toidentify a speed that minimizes time errors in the user data compared tothe record data, to adjust the record data according to the identifiedspeed, and to calculate the time errors of the user data based on theadjusted record data.
 8. The system of claim 1, wherein the processor isconfigured to identify timestamps in the user data and determine whetherthe start time and the end time of the user data are in accordance withthe record data based on the timestamps, to calculate the power of thebeats in the user data and analyze whether the power of the beats is ina correct pattern and a right timing, and to determine whether the speedof the user data has a trend of increasing or decreasing.
 9. The systemof claim 1, wherein the electronic display is configured to display theuser data and the record data on the same score line in an overlappedfashion or on two score lines in parallel with the same time scale, todisplay musical elements in the record data in a first type of color, todisplay musical elements in the user data with a strong power in asecond type of color, and to display musical elements in the user datawith a weak power in a third type of color.
 10. A computer-implementedmethod for music education, the method comprising: storing a set ofrecord data that contains information about speed, timing, frequency andbeats of a standard musical sample on a storage device; recording amusical exercise with a recording device while determining the start andthe end of the musical exercise by the detected volume of the musicalexercise with a processor; converting the recording of the musicalexercise to a set of user data with the processor; retrieving the set ofrecord data from the storage device and mapping the set of user data tothe set of record data with the processor; analyzing differences betweenthe user data and the record data based on the mapping and therebycalculating a series of values related to speed, timing, frequency orpower of the user data with the processor; calculating a score as thesum of the series of values multiplied by a series of coefficientsrespectively with the processor; and displaying the score and thedifferences between the user data and the record data to a user throughan electronic display; wherein recording the musical exercise comprisesstarting recording by the processor when the detected volume of themusical exercise has been higher than a predetermined threshold for apredetermined number of times, and ending recording by the processorwhen the detected volume of the musical exercise has been lower than apredetermined threshold for a predetermined number of times.
 11. Themethod of claim 10, wherein the musical exercise comprises a sequence ofclapping, and converting the recording of the musical exercise to theset of user data comprises calculating the sum of the peak power of theclapping multiplied by a first coefficient and the total power of theclapping multiplied by a second coefficient, and thereby detecting thesequence of clapping from the set of user data and eliminating noisefrom the result of the detection by the processor.
 12. The method ofclaim 11 further comprising calculating the duration of each clap in theadjusted user data and in the record data respectively, and determiningwhether the difference therebetween is within a predetermined toleranceby the processor.
 13. The method of claim 10, wherein the musicalexercise comprises singing or music instrument playing, and the methodfurther comprising filtering out any music note that is in apredetermined frequency in the user data and adjusting the scale of theuser data according to the scale of the record data before the mappingby the processor.
 14. The method of claim 13 further comprisingtransforming the record data and the user data into a matrix,calculating a cost of error for each node of the matrix, and adjustingthe speed and the key signature of the record data by finding a pathwith the lowest accumulative cost of error based on differentcombination of speeds and key signatures by the processor.
 15. Themethod of claim 13 further comprising marking a reference point in theuser data and adjusting the frequencies of all the music notes in theuser data based on the difference between the user data and the recorddata at the reference point by the processor.
 16. The method of claim 15further comprising calculating the frequency errors in the user databased on the mapping, identifying a speed that minimizes time errors inthe user data compared to the record data, adjusting the record dataaccording to the identified speed, and calculating the time errors ofthe user data based on the adjusted record data.
 17. The method of claim10 further comprising identifying timestamps in the user data anddetermining whether the start time and the end time of the user data arein accordance with the record data based on the timestamps, calculatingthe power of the beats in the user data and analyzing whether the powerof the beats is in a correct pattern and a right timing, and determiningwhether the speed of the user data has a trend of increasing ordecreasing by the processor.
 18. A computer-implemented method for musiceducation, the method comprising: storing a set of record data thatcontains information about speed, timing, frequency and beats of astandard musical sample on a storage device; recording a musicalexercise with a recording device; converting the recording of themusical exercise to a set of user data with a processor; retrieving theset of record data from the storage device and mapping the set of userdata to the set of record data with the processor; comparing the userdata to the record data based on the mapping with the processor; anddisplaying the difference between the user data and the record data aswell as a score representing a quantitative evaluation of the musicalexercise based on the comparison to a user through an electronicdisplay; wherein: musical elements in the record data is displayed in afirst type of color, musical elements in the user data with a strongpower is displayed in a second type of color, and musical elements inthe user data with a weak power is displayed in a third type of color;wherein recording the musical exercise comprises starting recording bythe processor when the detected volume of the musical exercise has beenhigher than a predetermined threshold for a predetermined number oftimes, and ending recording by the processor when the detected volume ofthe musical exercise has been lower than a predetermined threshold for apredetermined number of times.